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FOREWORD 


By Michael J. Petrilli and Amber M. Northern 


Those of us at Fordham have strived over the course of our organization's two-decade existence 
to stay open to new evidence and to be willing to change our minds. For example, we shifted 
from the "let a thousand flowers bloom" camp within the charter movement to the "some 
weeding is necessary" team after multiple studies showed just how poor the achievement 

of some charters was turning out to be, and just how hard it was to actually shut such failing 
charter schools down. And there have been other smaller shifts over the years too, on funding, 
teacher diversity, and more. 


Of course, we are fortunate to be an independent think tank, with our own endowment and a 
mission to follow the evidence wherever it leads. It's not so easy to keep an open mind when 
you're an interest group, like a teachers union, which is charged with protecting its members’ 
concerns. The unions are never going to say, "You know, we've changed our minds based on the 
evidence, and we've decided we really like these non-unionized charter schools." So for them 
the game is about finding evidence that supports their position and ignoring, if not discrediting, 
the rest. 


We see that on our side of the reform fence too, as when some of our friends decided that 
test scores weren't valid measures of student success after multiple rigorous studies showed 
negative test score impacts of private school choice programs. Thus the energetic effort to 
discredit test scores as predictors of long term outcomes. 


But back to reform opponents. There used to be a fierce debate about whether kids in charter 
schools were learning more than kids in traditional public schools. The American Federation 
of Teachers, for example, famously leaked a study to the New York Times that relied on NAEP 
scores to make this tendentious point. But now that dozens of studies have found that kids 

of color in urban charter schools learn significantly more on average than their district peers, 
the unions and other opponents have had to find other arguments to make their case, while 
working to discredit the impact evaluations. 


So they now have a two-part argument. 


First, they argue that the charter school advantage is due entirely to “creaming.” For example, in 
a recent Wall Street Journal article, the co-chairman of United Teachers Los Angeles attributed 
the success of charter schools to “having classes filled with motivated, high-performing 
students.” It's apparently not enough to claim that some of the advantage comes from 
selection—the likelihood that families who choose charters are different in important ways from 
those who don't. But all of it? 


Second, they argue that, regardless of how good or bad charters might be for the kids they 
serve, their growth is hurting traditional public schools and the kids who are left behind there. 
This argument has the benefit of polling extremely well, and has been used to great—or 
terrible—effect in the current California charter school wars. 
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With that long context in mind, we are pleased to present the following highly significant 
analysis by Fordham's senior research and policy associate, David Griffith, which examines the 
relationship between charter school market share and student achievement—not just for the 
kids in charters or the kids in district schools but for everybody. 


It sounds straightforward, but to our knowledge this is the first time anyone has conducted such 
a study. 


We have the CREDO evaluations and other studies looking at the performance of students in 
charter versus district schools. And there have been many studies of "competitive” or “spillover” 
effects of charter schools on district schools, most of which find that competition from charter 
schools does not harm achievement in nearby district schools, and sometimes boosts it. 


This study doesn't look at the differences between charter and district kids. In fact, we can't 
distinguish between them because our data source, the Stanford Education Data Archive 
(SEDA), includes the academic progress that a// students in a given geographic community 
made compared to students in other geographic communities nationwide. In other words, the 
academic performance of charter schools is included in what SEDA refers to as the “geographic 
school district,” regardless of whether the charter schools operate independently of the 

district in which they are located (although we do know what percentage of students attended 
charters). 


Griffith spent an entire year getting acquainted with the dataset and searching for the best 
way to model “charter market share.” For example, because we're really interested in the 
achievement of specific racial subgroups, it makes more sense to consider the effects of 
“charter market share” within those subgroups than across them. So his findings focus on 

the relationship between the percentage of black, white, or Hispanic students who enrolled in 
charters and the average achievement of all black, white, or Hispanic students in a geographic 
school district, including those in traditional public schools. 


In the end, what we learned was quite simple: For large urban districts, the more black and 
Hispanic students are enrolled in charter schools, the greater achievement is for black and 
Hispanic students. 


This has huge implications for the two arguments that charter opponents are making. 


First, it provides new evidence that creaming can't explain the entirety of the charter school 
advantage in urban districts. Because if charter schools' success was truly an "illusion," as the 
Wall Street Journal author charges, we'd see no gains for communities with greater charter 
market share. Higher test scores in charters would be canceled out by lower test scores in district 
schools, driven by the transfer of higher-achieving students from district to charter schools. It 
would be a zero-sum game. 


Instead, we find achievement gains in districts with more charters. What that implies is that the 
additional learning that's happening in charter schools is not coming at the expense of less 
learning in district schools. It's additive. And that implies that kids in charters really are learning 
more—not because of who the kids are, but because of what the schools are doing. 
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The findings also have implications for opponents’ argument that charter schools are hurting 
traditional public schools. We suspected that wasn't the case, based on existing evidence, at 
least for the district schools located closest to new charter schools, at least when it comes to test 
scores. But maybe the performance of other students in the district was being harmed in some 
way. We don't see any evidence of that in this study. 


Of course, this one analysis won't end the charter school wars, or even change them ina 
significant way. Perhaps some scholars or bystanders who mildly opposed charter schools will 
change their minds, now that there's even more evidence that they really are doing something 
to help black and Hispanic kids learn more, and that they aren't hurting the outcomes of 
students in district schools. Yet we suspect that most critics will continue to oppose charter 
schools because their opposition has always been based on bread and butter interests, like the 
bargaining power of teachers unions, rather than evidence or reason. 


But, charter supporters, stand proud. These schools really are getting better results for children 
of color, and not just because they are attracting the most motivated families. 


That's good news with which to kick off the new school year. 
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EXECUTIVE SUMMARY 


Although numerous studies have examined the relative performance of charter and traditional 
public schools, as well as the competitive effects that charters may have on their district 
neighbors, to our knowledge, no prior study has addressed whether overall achievement 
increases as the “market share” of charter schools rises. 


Accordingly, this report uses data from a new source, which allows researchers to compare 
English language arts (ELA) and math scores from thousands of school districts and dozens of 
different tests, to address two questions: 


|. Is higher charter market share associated with achievement gains for all students— 
including those enrolled in traditional (i.e., district-operated) public schools? 


2. How does the relationship between charter market share and achievement differ by 
district and student characteristics? 


Unlike previous studies, this one neither compares nor distinguishes between the achievement 
of students in charter and traditional public schools. Rather, it focuses on the overall 
performance of “geographic school districts,” which include both traditional public schools and 
local charters (even if they are legally and functionally independent). Furthermore, the effects of 
higher “charter market share” are examined within racial subgroups rather than across them. In 
other words, the findings focus on the relationship between the percentage of black, white, or 
Hispanic students who enrolled in charters and the average achievement of all black, white, or 
Hispanic students in a geographic school district, including those in traditional public schools. 
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FINDING 1: In urban areas, higher charter market share is associated 
with significant achievement gains for black and Hispanic students. 


In major urban areas, higher charter market share among black and Hispanic students is 
associated with significant achievement gains in both ELA and math. For example, in the 
twenty-one urban districts with the most black students, moving from 0 to 50 percent “black 
charter market share” is associated with a 0.8 grade level increase in average ELA achievement 
and a 0.7 grade level increase in average math achievement for all black students—including 
those in traditional public schools (Figures ES.1-ES.2). 


Figures ES.1—-ES.2. In the largest urban districts, higher “black charter market 
share” is associated with significant achievement gains for black students. 


Predicted achievement gains 


} Fig. ES. Predicted achievement gains | Fig. ES.2 | 
sazecses 90% confidence interval -------- 90% confidence interval 


N = 728 Units = 121 Districts = 21 N = 686 Units = 121 Districts = 21 


Change in Average Black ELA 
Achievement (in Grade Levels) 
Change in Average Black Math 
Achievement (in Grade Levels) 


0% 10% 20% 30% 40% 50% 0% 10% 20% 30% 40% 50% 
Percentage of Black Students Percentage of Black Students 
Enrolled in Charter Schools Enrolled in Charter Schools 


Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 50 percent black charter market share and average black enrollment >2500 between 2009 and 2015. 
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Similarly, in the twenty-seven urban districts that enroll the most Hispanic students, moving 
from 0 to 35 percent “Hispanic charter market share” is associated with a 0.7 grade level increase 
in average ELA and math achievement among Hispanic students—though because Hispanic 
charter market share didn’t exceed 35 percent in any of these districts, we can't estimate the 
achievement gains beyond this point (Figures ES.3—-ES.4). 


Figures ES.3—-ES.4. In the largest urban districts, higher “Hispanic charter market 
share” is associated with significant achievement gains for Hispanic students. 


| Fig. ES. | Predicted achievement gains | Fig. E54 
aca 90% confidence interval Renesas 


N = 829 Units = 145 Districts = 27 N = 703 Units = 143 Districts = 27 


Predicted achievement gains 


90% confidence interval 


Change in Average Hispanic ELA 
Achievement (in Grade Levels) 
Achievement (in Grade Levels) 


0% 10% 20% 30% 40% 50% 0% 10% 20% 30% 40% 50% 


Change in Average Hispanic Math 


Percentage of Hispanic Students Percentage of Hispanic Students 
Enrolled in Charter Schools Enrolled in Charter Schools 


Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 Hispanic students in charter 
and traditional public schools as district-by-grade-level charter market share increases among Hispanic students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 35 percent Hispanic charter market share and average Hispanic enrollment >2500 between 2009 and 2015. 
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FINDING 2: In suburban and rural areas, higher charter market share is 
associated with significant achievement gains for Hispanic students, 
and black students in rural districts also see gains. 


For example, in suburban districts with at least 500 Hispanic students per grade level, moving 
from 0 to 50 percent Hispanic charter market share is associated with a gain of 0.5 grade levels 
in ELA for Hispanic students—although there is no evidence that Hispanic students in suburban 
areas see gains in math (Figures ES.5—ES.6). 


Figures ES.5-ES.6. In large suburban districts, higher "Hispanic charter market 
share" is associated with significant ELA gains for Hispanic students. 


| Fig. ES.5 | Predicted achievement gains | Fig. ES.6 
Sees 90% confidence interval 


N = 3,061 Units = 641 Districts = 126 N = 2,603 Units = 613 Districts = 124 


Predicted achievement gains 


mereegs: 90% confidence interval 


Achievement (in Grade Levels) 
Achievement (in Grade Levels) 


0% 10% 20% 30% 40% 50% 0% 10% 20% 30% 40% 50% 


Change in Average Hispanic ELA 
Change in Average Hispanic Math 


Percentage of Hispanic Students Percentage of Hispanic Students 
Enrolled in Charter Schools Enrolled in Charter Schools 


Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 Hispanic students in charter 
and traditional public schools as district-by-grade-level charter market share increases among Hispanic students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
suburban units with 0 to 50 percent Hispanic charter market share and average Hispanic enrollment >500 between 2009 and 2015. 
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Similarly, higher black charter market share in rural districts is associated with significant 
ELA gains for black students in these communities (Figures ES.7-ES.8). However, there is no 
evidence that higher black charter market share in suburban districts benefits black students 
(not shown). 


Figures ES.7—-ES.8. In rural districts, higher “black charter market share” is 
associated with significant gains in ELA achievement for black students. 


Fig. ES.7 Predicted achievement gains | Fig. ES.8 Predicted achievement gains 
a------- 90% confidence interval -------- 90% confidence interval 


N = 2,007 Units = 481 Districts = 105 N = 1,935 Units = 471 Districts = 104 


Change in Average Black ELA 
Achievement (in Grade Levels) 
Change in Average Black Math 
Achievement (in Grade Levels) 


0% 10% 20% 30% 40% 50% 0% 10% 20% 30% 40% 50% 
Percentage of Black Students Percentage of Black Students 
Enrolled in Charter Schools Enrolled in Charter Schools 


Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
rural units with 0 to 50 percent black charter market share and average black enrollment >100 between 2009 and 2015. 
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FINDING 3: There is no evidence that higher charter market share is 
associated with achievement gains for white students. 


For example, there is no evidence that higher "white charter market share" boosts the ELA or 
math achievement of white students in larger urban districts (Figures ES.9-ES.10). 


Figures ES.9-ES.10. In large urban districts, there is no significant relationship 
between "white charter market share" and white students’ ELA and math 
achievement. 


| Fig. ES.9 | Predicted achievement gains Fig. £5.10, 
pee? 90% confidence interval 


N = 1,753 Units = 360 Districts = 70 = 1,631 Units = 350 Districts = 70 


Predicted achievement gains 


Bee 90% confidence interval 


Change in Average White ELA 
Achievement (in Grade Levels) 


Achievement (in Grade Levels) 
fo} 
ro} 
| 


0% 10% 20% 30% 40% 50% 


Change in Average White Math 


0% 10% 20% 30% 40% 50% 


Percentage of White Students Percentage of White Students 
Enrolled in Charter Schools Enrolled in Charter Schools 


Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 white students in charter and 
traditional public schools as district-by-grade-level charter market share increases among white students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 50 percent white charter market share and average white enrollment >1000 between 2009 and 2015. 
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Similarly, there is no evidence that white higher charter market share boosts the academic 
achievement of white students in suburban districts (not shown) or in rural areas—although it 
doesn't seem to do much harm in these places either (Figures ES.11-ES.12). 


Figures ES.11-ES.12. In rural districts, there is no significant relationship between 
"white charter market share" and white students’ ELA and math achievement. 


| Fig. 5.1 | Predicted achievement gains Fig. £5.12, 
ae 90% confidence interval 


N = 3,366 Units = 803 Districts = 169 N = 3,210 Units = 785 Districts = 167 


Predicted achievement gains 


Sete snot 90% confidence interval 


Change in Average White ELA 
Achievement (in Grade Levels) 
Change in Average White Math 
Achievement (in Grade Levels) 


0% 10% 20% 30% 40% 50% 0% 10% 20% 30% 40% 50% 


Percentage of White Students Percentage of White Students 
Enrolled in Charter Schools Enrolled in Charter Schools 


Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 white students in charter and 
traditional public schools as district-by-grade-level charter market share increases among white students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
rural units with 0 to 50 percent white charter market share and average white enrollment >250 between 2009 and 2015. 


In short, when it comes to white charter market share, there simply isn't much to talk about. 


In contrast, the academic gains associated with higher black and Hispanic charter market share 
are substantial and highly consistent with the large, overwhelmingly positive literature on 

the performance of charters that serve these groups, as well as the smaller, but still positive, 
literature on charters’ competitive effects. 


Obviously, the logical implication of these results is that increasing the percentage of black and 
Hispanic students who enroll in charters—especially in the largest urban districts, which educate 
millions of minority students per year—would significantly reduce the longstanding racial 
achievement gaps that are ostensibly of concern to policymakers. 


13 


I. Introduction | Rising Tide: Charter School Market Share and Student Achievement 


l. INTRODUCTION 


For education reformers, it’s a trillion-dollar question: Does a rising tide of charter schools lift all 
education boats? 


In other words, are charters really boosting the overall performance of our K-12 education 
system—at least in most places, and especially for low-income students and students of color— 
as research suggests? Or are the mischievous education seas once again playing tricks on 
unsuspecting reformers? For example, might charters’ apparent success actually be coming at 
the expense of traditional public schools—perhaps via the “creaming” of their best students or 
perhaps through some other mechanism—leading to the illusion of overall progress where in 
fact none has occurred? 


From the helm of a sinking vessel, even a trough can look like a swell. But that doesn't mean the 
ocean is rising. Similarly, no matter what statistical wizardry researchers employ, comparisons 
between charter and traditional public schools within education markets can never fully capture 
the former's overall effect on academic achievement—as sophisticated critics of the charter 
movement are quick to note. 


Making any headway on that analytic front requires comparisons between and across education 
markets—or at least between and across reasonably large school districts—with varying levels 
of charter penetration. Yet, even in larger states, there aren't that many big school districts with 
consequential numbers of charters in their midst. And a federal system that allows states to 
design their own English language arts (ELA) and math assessments makes it tough to compare 
the performance of districts across state lines. 


Tough, but not impossible—thanks to the Stanford Education Data Archive (SEDA), a new data 
source that uses states’ performance on the National Assessment of Education Progress (NAEP) 
to make comparable ELA and math scores from dozens of different state tests and thousands of 
school districts. 


Accordingly, this report seeks answers to two fundamental but inherently difficult questions: 


|. Is higher charter market share associated with achievement gains for all students— 
including those enrolled in traditional (i.e., district-operated) public schools? 


2. How does the relationship between charter market share and achievement differ by 
district and student characteristics? 


Although there are no randomized controlled trials when it comes to higher “charter market 
share,” the answers to these questions are vitally important—and, as demonstrated by the 
findings, definitive enough to point state and local policymakers in the right direction. 


Obviously, there would be cause for concern if communities with a growing charter presence 
were experiencing declining performance—even if the reasons for such decline weren't clear. 
So what is the appropriate reaction or course of action if the opposite is true—that is, if all 
seaworthy vessels, regardless of their flag, seem to be rising along with the charter tide? 
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ll. BACKGROUND 


Although the subject of “charter market share” intersects with many lines of research, two 
literatures are particularly relevant to the present study. 


First, there have been numerous attempts to gauge the relative performance of charter and 
traditional public schools in ELA and math. For example, many early studies used the admissions 
lotteries conducted by oversubscribed charters to compare the performance of students who 
were and were not admitted to these schools, with many (though by no means all)' finding 
positive effects,” especially for students of color.* However, since such lotteries are only held by 
relatively popular (and presumptively higher-performing) charters, concerns were raised about 
the generalizability of these studies. 


Accordingly, for the past decade, Stanford's Center for Research on Education Outcomes 
(CREDO) has collected student-level data from dozens of states in an effort to summarize 
the relative performance of the charter sector. For example, a 2009 study found that students 
in charter schools made less progress in ELA and math, on average, than otherwise similar 
students in traditional public schools.* However, by 2013, CREDO's estimates suggested that 
charters had edged past their district counterparts in ELA, while essentially pulling even in 
math.° And a subsequent analysis of charter performance in major urban areas found large 
positive effects in both subjects—especially for low-income students and (once again) for 
students of color.® 


Less numerous (but no less important) than the studies of charters’ relative performance are the 
dozen or so empirically rigorous studies that have examined their impact on traditional public 
schools, most of which have found neutral or slightly positive effects.’ For example, a 2013 study 
of charter competition in North Carolina found positive effects on ELA and math achievement in 
traditional public schools, which were concentrated in “overlapping” grade levels.® Similarly, a 
2014 study found that traditional public schools in the District of Columbia improved in response 
to competition from higher-quality charters—but not in response to higher market share 
(defined as the percentage of students who transferred to neighboring charters).? Finally, a 2016 
study found that charters in New York City had positive “spillover effects” on district schools, 
which increased with their physical proximity.” 


In combination, the literature on charters’ performance relative to traditional public schools and 
the literature on charters’ competitive effects suggest that the “equilibrium effects” of higher 
charter market share (that is, its overall effects on the academic achievement of al// students in 
community) are positive—at least in most locations (and for most students). However, to our 
knowledge, no prior study has directly addressed this trillion-dollar question. 
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Ill. DATA 


Our primary data source for this project is the Stanford Education Data Archive (SEDA), which 
includes estimates of average ELA and math achievement for more than 13,000 geographic 
school districts and more than 80,000 district-by-grade-level units (e.g., fourth graders in the 
Los Angeles Unified School District). Collectively, these data span six grade levels (grades 3-8) 
and seven years (2009 through 2015). However, because many states changed tests at some 
point during the study period, many districts and district-by-grade-level units are missing data 
for at least one year (resulting in an unbalanced panel)." 


In addition to estimates of average district-by-grade-level achievement, SEDA also includes 
data on a host of district and/or district-by-grade-level covariates (e.g., the percentage of 
students who are English language learners). Although most of these data were originally 
assembled from other sources, because some smaller school districts didn't always report their 
enrollment data to the National Center for Education Statistics (NCES), some of the covariate 
data in SEDA have been imputed. 


Importantly, because SEDA assigns individual charter campuses to “geographic school districts” 
based on their physical addresses, the academic performance of local charters is included 

in SEDA's estimates of average district-by-grade-level achievement, even when the charters 

in question are legally and functionally independent from the surrounding district. In other 
words, the dependent variable—average district-by-grade-level achievement—reflects the 
performance of both students in traditional public schools and students in charter schools. 


|" 


Similarly, the variable of interest—district-by-grade-level “charter market share”—includes 
both students who attended district-authorized charters (where such exist) and students who 
attended other local charters within the boundaries of the relevant district. However, although 
we relied on SEDA for data on achievement and covariates, because the current version of SEDA 
doesn't include district-by-grade-level data on charter enrollment, our “charter market share” 
variables were constructed by downloading the relevant school-by-grade-level enrollment 

data from the National Center for Education Statistics (NCES) and merging them with the SEDA 
data.” 


Although the ultimate goal is to characterize the effects of higher charter market share at the 
district level, for several reasons, it makes more sense to analyze these effects at the "district- 
by-grade" level: First, insofar as the effects of higher charter market share are attributable to 
charters themselves, it makes no sense to estimate the effect of higher market share in grades 
6-8 on the ELA and math achievement of students in grades 3-5. Second, prior research 
suggests that any competitive effects that charters have on traditional public schools are 
strongest in “overlapping” grade levels (i.e., within district-by-grade-level units). Finally, looking 
at each district-by-grade-level unit separately gives us a useful source of variation that is 
plausibly independent of district-level trends (see Appendix A). 
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IV. SAMPLE 


Because our analysis focuses on the changes in charter market share that district-by-grade-level 
units experienced over time, we restricted the sample to those units with at least some charter 
market share during the study period—and in the case of our subgroup estimates (which are 
the basis for our findings), at least some black, white, or Hispanic charter market share. Similarly, 
because very few large districts had charter market share that exceeded 50 percent—either 
overall or for the three biggest racial subgroups—we decided to exclude districts that exceeded 
this threshold, rather than present estimates for the 50 to 100 percent range that would 
necessarily depend on an inappropriate level of extrapolation.” 


In addition to these cuts, we also excluded any district-by-grade-level units with fewer than 

100 students in the relevant subgroup—that is, with fewer than 100 black, white, or Hispanic 
students. Although these thresholds are arbitrary, limiting the sample in this manner is helpful 
insofar as it reduces the influence of very small districts and districts with very few students 

in the subgroup of interest, as well as the potentially confounding effect of unobserved 
interdistrict choice (see Limitations). 


In practice, district-by-grade units with at least 100 students in the relevant subgroup— 
essentially districts with at least 100 students in these groups per grade level—enrolled more 
than 95 percent of students in units with non-zero charter market share. In other words, if our 
ultimate goal is to characterize the overall effects of higher charter market share for the average 
student, as opposed to the average district, we lose very little by restricting the sample in the 
manner described—especially in urban districts (which are larger on average) and for black and 
Hispanic students (for whom charter attendance is particularly concentrated in urban areas). 
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V. METHODS 


To isolate the relationship between district-by-grade-level charter market share and average 
district-by-grade-level achievement in ELA and math, we rely on a combination of demographic 
controls, district-by-grade-level and year fixed-effects, and district-specific time-trends (see 
Appendix A for more). In a nutshell, our model asks whether students in grade levels where 
charter market share increased during the study period saw a larger or smaller increase in 
average ELA and math achievement than one would expect given the broader (i.e., district-level) 
achievement trends in those subjects and the observable changes in student demographics. 
However, because the effect of “higher charter market share” is actually the product of many 
different effects—from the transition costs associated with changing schools, to the short-term 
costs of closing a low-performing school—we can't assume that its relationship to academic 
achievement is strictly linear, or that the returns to higher market share increase or decrease 
consistently. Accordingly, rather than assuming linearity, we decided to “let the data speak” via 
a series of cubic “splines” that allow the slope and curve of the function to change at the 5th, 
35th, 65th, and 95th percentiles of charter market share.” 


For example, Figures 1-2 show the relationship between tota/ charter market share in district- 
by-grade-level units with at least 500 students and the average achievement of al! students— 
including those in traditional public schools. 


Figures 1-2. Higher charter market share among "all students" is associated with 
a significant increase in ELA achievement. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 students in charter and 
traditional public schools as district-by-grade-level charter market share increases among all students. Estimates were generated 
using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student demographics, 
district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all units with 0 to 
50 percent charter market share and average enrollment >500 between 2009 and 2015. 
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Per the figures, each spline is accompanied by a 90 percent confidence interval, allowing the 
reader to gauge statistical significance. For example, Figure 1 implies that moving from 0 to 50 
percent charter market share is associated with a statistically significant increase in overall ELA 
achievement, which is roughly equivalent to 0.2 grade levels.'® However, per Figure 2, there is 
no statistically significant relationship between total charter market share and average math 
achievement—consistent with contemporary estimates of charters’ relative performance.” 


Similarly, there is evidence of a positive relationship between total charter market share and the 
average ELA achievement of black, white, and Hispanic students, including those in traditional 
public schools. For example, in districts with at least 500 students and at least 100 black, white, 
or Hispanic students, moving from 0 to 50 percent charter market share (among all students) 

is associated with a gain of 0.3 grade levels for black students, as well as (suggestive) gains 

of 0.2 grade levels for Hispanic students and 0.1 grade levels for white students (Figures 3—5). 
However, although it may seem intuitive, this is probably not the best way to think about charter 
market share. For example, total charter market share and average white achievement are 
positively correlated in the District of Columbia (where charters’ growth has coincided with rapid 
gentrification). Yet it seems unlikely that charters in D.C. have had a significant impact on white 
students given how few of them enroll. (And for similar reasons, the fact that higher overall 
charter market share was associated with achievement gains for all District of Columbia students 
isn't as illuminating as it might be.) 


As that example suggests, if we're really interested in the impact of “charter market share” on 
the achievement of specific racial subgroups, it makes more sense to consider the effects of 
“charter market share” within those subgroups than across them. Accordingly, the findings that 
follow focus on the relationships between higher charter market share within black, white, and 
Hispanic communities—that is, the percentage of black, or white, or Hispanic students that 
enrolled in charters—and the overall achievement of students in those groups—that is, the 
average achievement of all black, white, or Hispanic students in a geographic school district, 
including those in traditional public schools. 
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Figures 3-5. Higher charter market share among "all students" is associated with 
a significant increase in black students' ELA achievement (and there is suggestive 
evidence of ELA gains for Hispanic students). 
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Per Table 1, roughly three-quarters of black students and three-fifths of Hispanic students who 
enrolled in a charter school between 2009 and 2015 were in districts that NCES classifies as 
“urban,” as were at least three-fifths of a/! black and Hispanic students in districts and grades 
with non-zero black and Hispanic market share—that is, in the markets and communities that 
were affected by the schools in question (Table 2). Consequently, we begin with a discussion 
of the relationship between higher black and Hispanic charter market share and the average 
achievement of students in these groups in major urban locations. 


Table 1. Distribution of black, Hispanic, and white charter school enrollment by 
district type (i.e., urban, suburban, and rural). 


Hispanic White All Students 


Urban 74% 62% 
Suburban 22% 30% 


Rural 4% 7% 


All Districts 100% 100% 100% 100% 


Note: Rural districts include those classified as "Town" by the National Center for Education 
Statistics. Due to the volume of missing and/or imputed data, percentages are approximate. 


Table 2. Distribution of black, Hispanic, and white students whose communities are 
affected by charter schools by district type (i.e., urban, suburban, and rural). 


Hispanic White All Students 
Urban 63% 61% 52% 
Suburban 32% 34% 39% 


Rural 5% 5% 


9% 
All Districts 100% 100% 100% 100% 


Note: Rural districts include those classified as "Town" by the National Center for Education 
Statistics. Due to the volume of missing and/or imputed data, percentages are approximate. 
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VI. FINDINGS 


FINDING 1: In urban areas, higher charter market share is associated 
with significant achievement gains for black and Hispanic students. 


In urban areas, higher charter market share among black students is associated with significant 
achievement gains for black students. For example, in urban districts with at least 100 black 
students per grade level, our estimates imply that moving from 0 to 50 percent “black charter 
market share” is associated with a gain of 0.3 grade levels in ELA and 0.4 grade levels in math— 
for all black students, including those in traditional public schools (Figures 6-7)."® 


Figures 6-7. In smaller urban districts, higher “black charter market share” is 
associated with significant achievement gains for black students. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 50 percent black charter market share and average black enrollment >100 between 2009 and 2015. 
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Similarly, among urban districts with at least 100 Hispanic students per grade level, moving from 
0 to 50 percent “Hispanic charter market share” is associated with a gain of 0.3 grade levels in 
math—for all Hispanic students, including those in traditional public schools—although there is 
no evidence that Hispanic students see gains in ELA (Figures 8-9). 


Figures 8-9. In smaller urban districts, higher “Hispanic charter market share” is 
associated with a significant increase in math achievement for Hispanic students. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 Hispanic students in charter 
and traditional public schools as district-by-grade-level charter market share increases among Hispanic students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 50 percent Hispanic charter market share and average Hispanic enrollment >100 between 2009 and 2015. 


In general, these estimates of the effects of higher black and Hispanic charter market share on 
overall black and Hispanic achievement in urban districts are consistent with prior estimates 
of the relative performance of urban charters (see Appendix B). However, because the size of 
urban districts varies so dramatically, they may actually understate the gains for the average 
black or Hispanic student in these places. 
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For example, if we restrict the sample to the twenty-one urban districts with at least 2,500 black 
students per grade level (see Appendix C, Table C.1), our estimates imply that moving from 

0 to 50 percent black charter market share is associated with gains of 0.8 grade levels in ELA 
and 0.7 grade levels in math for black students (Figures 10-11). In other words, the relationship 
between "black charter market share” and black students’ ELA and math achievement is even 
more positive for these very large districts, which collectively accounted for over half of black 
enrollment in urban districts where at least one black student enrolled in a charter. 


Figures 10-11. In the largest urban districts, higher “black charter market share” is 
associated with significant achievement gains for black students. 


Predicted achievement gains 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 50 percent black charter market share and average black enrollment >2500 between 2009 and 2015. 
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Similarly, in the twenty-seven urban districts with at least 2,500 Hispanic students per grade 
level (see Appendix C, Table C.2), our estimates imply that moving from 0 to 35 percent Hispanic 
charter market share is associated with a 0.7 grade level increase in average ELA and math 
achievement among Hispanic students (Figures 12-13). Unfortunately, because Hispanic charter 
market share didn't exceed 35 percent in any of these districts during the study period, we can’t 
be sure that moving beyond this point yields additional gains, although this seems extremely 
likely based on the shapes of our graphs. 


Figures 12-13. In the largest urban districts, higher “Hispanic charter market share” 
is associated with significant achievement gains for Hispanic students. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 Hispanic students in charter 
and traditional public schools as district-by-grade-level charter market share increases among Hispanic students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 35 percent Hispanic charter market share and average Hispanic enrollment >2500 between 2009 and 2015. 
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Notably, our estimates for Hispanic students are essentially unchanged if we include the ten 
large “suburban” districts that meet our enrollment criterion (Figures 14-15), some of which 
have a decidedly urban reputation despite being classified as suburbs by the National Center 
for Education Statistics (see Appendix C, Table C.3). 


Collectively, the thirty-seven districts in this larger group enrolled 45 percent of Hispanic 
students in communities where at least one Hispanic student attended a charter school. 
(See Finding 2 for a discussion of black charter market share in suburban districts.) 


Figures 14-15. In the largest urban and suburban districts, higher "Hispanic charter 
market share" is associated with significant gains for Hispanic students. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 Hispanic students in charter 
and traditional public schools as district-by-grade-level charter market share increases among Hispanic students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
units with 0 to 35 percent Hispanic charter market share and average Hispanic enrollment >2500 between 2009 and 2015. 


In sum, higher black charter market in urban districts is associated with significant achievement 
gains for black students—especially in the /Jargest urban districts, which enroll the majority of 
these students. Similarly, higher Hispanic charter market share in urban districts is associated 
with significant math gains for Hispanic students. And in the districts that enroll the most 
Hispanic students, it is associated with significant achievement gains in both subjects. 
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FINDING 2: In suburban and rural areas, higher charter market share is 
associated with significant achievement gains for Hispanic students, 
and black students in rural districts also see gains. 


Nearly a quarter of black students and a third of Hispanic students who enrolled in a charter 
school between 2009 and 2015 were in districts that NCES classifies as suburban, as were roughly 
a third of all black and Hispanic students in the affected districts. However, because charter 
penetration is lower in suburban areas, our estimates for these districts should be interpreted 
cautiously—especially when it comes to black students. 


In general, there is no evidence that higher black charter market share in suburban districts is 
associated with significant achievement gains (or losses) for black students, even when we limit 
the sample to larger districts. However, since only six suburban districts with >500 black students 
had a black charter market share that exceeded 25 percent between 2009 and 2015, it’s difficult 
to draw any firm conclusions about the effects of higher black charter market share in suburban 
areas, except that the outlook isn't as positive as it is in urban areas (Figures 16-17). 


Figures 16-17. In large suburban districts, higher "black charter market share" is not 
associated with significant achievement gains for black students. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
suburban units with 0 to 50 percent black charter market share and average black enrollment >500 between 2009 and 2015. 
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In contrast, among suburban districts with at least 500 Hispanic students per grade, which 
enrolled over three-quarters of Hispanic students in suburbs where one or more Hispanic 
students enrolled in a charter school, our estimates imply that moving from 0 to 50 percent 
Hispanic charter market share is associated with a gain of 0.5 grade levels in ELA but not in 
math. Though again, Hispanic charter market share exceeded 25 percent between 2009 and 2015 
in just nine of these districts (Figures 18-19). 


Per the discussion in Finding 1, the distinction between “urban” and “suburban” districts can be 
murky—especially in Florida, where there are a great many Hispanic students. But regardless of 
how these larger “suburban” districts are classified, higher charter market share in larger urban 
and suburban districts seems to be benefitting Hispanic students. 


Figures 18-19. In large suburban districts, higher "Hispanic charter market share" is 
associated with significant ELA gains for Hispanic students. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 Hispanic students in charter 
and traditional public schools as district-by-grade-level charter market share increases among Hispanic students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
suburban units with 0 to 50 percent Hispanic charter market share and average Hispanic enrollment >500 between 2009 and 2015. 
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Although rural charter schools are seldom associated with students of color, collectively they 
enrolled approximately 10,000 black students and 20,000 Hispanic students per year between 
2009 and 2015." Moreover, because the average rural district is much smaller than the average 
suburban district, those rural districts that do have charters tend to have higher “charter market 
share” (making them easier to analyze). 


In general, the evidence suggests that higher charter market share is associated with significant 
achievement gains for both black and Hispanic students in rural areas. For example, among 
rural districts with at least 100 black students per grade level, our estimates imply that moving 
from 0 to 50 percent black charter market share is associated with gains of 0.7 grade levels in 
ELA and 0.4 grade levels in math, although our math estimates never quite achieve statistical 
significance (Figures 20-21).”° 


Figures 20-21. In rural districts, higher “black charter market share” is associated 
with significant gains in ELA achievement for black students. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
rural units with 0 to 50 percent black charter market share and average black enrollment >100 between 2009 and 2015. 
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Similarly, our estimates for rural districts with at least 100 Hispanic students per grade level are 
positive but statistically insignificant (not shown). However, the evidence becomes clearer when 
we restrict the sample to include rural districts with at least 200 Hispanic students per grade, 
which still account for 60 percent of Hispanic enrollment in rural areas with charters (Figures 
22-23). For example, our estimates imply that moving from 0 to 50 percent Hispanic charter 
market share in these places is associated with a gain of 0.9 grade levels in ELA and 1.0 grade 
levels in math. 


Figures 22-23. In rural districts, higher "Hispanic charter market share" is 
associated with significant achievement gains for Hispanic students. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 Hispanic students in charter 
and traditional public schools as district-by-grade-level charter market share increases among Hispanic students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
rural units with 0 to 50 percent Hispanic charter market share and average Hispanic enrollment >200 between 2009 and 2015. 


In sum, higher black and Hispanic charter market share in rural districts is associated with 
significant achievement gains for black and Hispanic students. However, there is no evidence 
that higher black charter market share in suburban districts benefits black students in these 
places. Finally, Hispanic students in suburban districts see significant achievement gains in ELA, 
but not in math. 
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FINDING 3: There is no evidence that higher charter market share is 
associated with achievement gains for white students. 


Regardless of district size or location, there is no significant relationship between white charter 
market share and white students’ overall achievement in ELA and math (Figures 24-29), 
although there is suggestive evidence of a negative relationship in urban and suburban areas 
(Figures 24-27). Broadly speaking, this is consistent with prior research that suggests charters 
have a slightly negative effect on white students’ academic progress in major urban areas” as 
well as overall—especially in math.” 


Figures 24-25. In large urban districts, there is no significant relationship between 
"white charter market share" and white students’ ELA and math achievement. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 white students in charter and 
traditional public schools as district-by- grade-level charter market share increases among white students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 50 percent white charter market share and average white enrollment >1000 between 2009 and 2015. 
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Figures 26-27. In suburban districts, there is no significant relationship between 
"white charter market share" and white students’ ELA and math achievement. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 white students in charter and 
traditional public schools as district-by- grade-level charter market share increases among white students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
suburban units with 0 to 50 percent white charter market share and average white enrollment >500 between 2009 and 2015. 


Figures 28-29. In rural districts, there is no significant relationship between "white 
charter market share" and white students’ ELA and math achievement. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 white students in charter and 
traditional public schools as district-by- grade-level charter market share increases among white students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
rural units with 0 to 50 percent white charter market share and average white enrollment >250 between 2009 and 2015. 
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VII. LIMITATIONS 


Perhaps the biggest limitation of our study is that charter schools aren't randomly assigned 

to geographic school districts—and are, in fact, disproportionately likely to cluster in lower- 
performing districts that serve higher percentages of poor and minority students. As discussed 
in Appendix A, this pattern is like an econometric riptide, dragging our estimates down for 
reasons that have nothing to do with charters’ performance or their impact on traditional public 
schools. Consequently, insofar as our efforts to control for student demographics and other 
confounding factors fall short, we may be underestimating the gains associated with higher 
charter market share. 


Similarly, because we are working with just seven years of data, we may not be fully capturing 
the long-term benefits (or costs) of higher charter market share. For example, research 

suggests that students who switch schools experience an initial decline in achievement, and 
many charters open multiple grades simultaneously.”% So in grades where few students switch 
schools (e.g., grades 3 or 4), higher charter market share may be inevitably associated with lower 
achievement in the short run—that is, until the data reflect the experiences of later cohorts. 
Worse, if a low-performing charter closes during the study period, this may actually make 

the impact of “higher charter market share” look more negative, since our model effectively 
compares districts to themselves (and since we may not observe the opening of presumptively 
higher-performing charters that replace the school that closes).”* 


On a related note, it's also important to recognize that our estimates of the likely gains 
associated with "moving from 0 to 50 percent higher charter market share" are based on the 
collective experiences of hundreds of school districts and thousands of district-by-grade-level 
units, most of which didn't actually experience such a large increase in charter market share 
during the study period. In other words, a fundamental assumption of our model is that the 
modest increases in charter market share (and the commensurately modest achievement gains) 
that districts on different (but often overlapping) parts of the charter market share continuum 
experienced during the study period can be combined into a single estimate of the average 
district's expected achievement gains over the entirety of that continuum. 


Finally, without student-level data, we can't fully account for the movement of individual 
students across district boundaries, which could potentially bias our estimates in either 
direction. However, if the achievement gains that we attribute to higher charter market share 
were actually driven by the unobserved movement of high-achieving charter students across 
district lines, we would expect the slopes of our graphs to decrease as district size increases, 
which is the opposite of what we observe. In other words, it’s possible that we are (again) 
underestimating the benefits of charters due to unobserved student mobility, at least for black 
and Hispanic students in smaller urban districts.”° 
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VIII. TAKEAWAYS 


So does a rising tide of charters lift all education boats? 


Although reality is rarely as simple as a good metaphor, our findings suggest that the short 
answer is “yes"—at least for black students in urban areas and especially for Hispanic 
students—though not, as far as we can tell, for white students. 


In our view, these findings have at least three implications for policy, none of which should come 
as a surprise to observers who are familiar with the prior research on charter performance and 
competition: 


1. Student achievement isn't a zero-sum game. 


In general, our results suggest that charters really are boosting the achievement of black and 
Hispanic students, rather than “creaming” the best students in these communities. Moreover, 
although we can't disentangle the effects of charter competition from the effects of actually 
attending a charter school, our results are highly consistent with prior estimates of charters’ 
relative performance—suggesting that most of the gains that black and Hispanic students 
experience are driven by charter attendance. 


2. There is little evidence that charters have a negative effect on 
traditional public schools. 


Again, we can't disentangle the effects of charter competition from the effects of actually 
attending a charter school. (So, in theory, either one of these effects could be negative if 
the other was sufficiently positive.) However, when viewed alongside CREDO’'s estimates of 
charters’ relative performance, our estimates for black and Hispanic students imply a neutral 
(or perhaps slightly positive) effect on the performance of traditional public schools in these 
communities (see Appendix B). 


3. Expanding charter market share in black and Hispanic communities 
could dramatically reduce racial achievement gaps. 


Although we can't precisely quantify the benefits of higher charter market share, our estimates 
suggest they are substantial for students of color, who currently trail white students by two to 
three grade levels when it comes to reading, arithmetic, and other prerequisites of upward 
mobility and effective citizenship. 


In light of this evidence and the innumerable other studies that suggest enrolling in a charter 
is associated with substantial achievement gains for black and Hispanic students, it is simply 
wrong to stand in the way of charters’ continued growth in these communities if closing racial 
achievement gaps is truly the goal. 


Henceforth, the burden of proof falls on those who would claim otherwise. 
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APPENDIX A 


TECHNICAL APPENDIX 


To isolate the relationship between district-by-grade-level charter market share and average 
district-by-grade-level achievement in ELA and math, we rely on a combination of demographic 
controls, district-by-grade-level and year fixed-effects, and district-specific time-trends. 


Specifically, we control for the percentage of students who are white, black, Hispanic, Asian, 
or eligible for a free or reduced-priced lunch in a given district-by-grade-level unit and year. 
Similarly, we control for the percentage of students in the average white, black, or Hispanic 
student's school who are eligible for federal lunch subsidies (since we don't have data on 
the percentage of students in these groups who are so eligible). Finally, we control for the 
percentage of students who are English language learners or eligible for special education 
services in a given district and year (since these data aren't collected at the district-by-grade 
level). 


In addition to these observables, we also control for any unobserved year-specific shocks 

insofar as they are common to all district-by-grade-level units through the inclusion of year- 
specific dummy variables. (For example, we control for the effects of the recession in 2010, 
insofar as they were common to all units.) Similarly, we control for any unobserved time- 
invariant differences between district-by-grade-level units through the inclusion of unit-specific 
dummies. (For example, we control for the differences between fourth graders in Chicago and 
seventh graders in Martha's Vineyard, insofar as they remained constant over the course of the 
study period.) 


Finally, since it’s likely that academic achievement in most districts was increasing or decreasing 
for reasons that had nothing to do with charter schools and may not be fully captured by our 
demographic controls, we include a quadratic “time-trend” for every district in our sample. In 
other words, insofar as the various district-by-grade-level units in a district were trending in the 
same direction, we control for that trend. (And since the implication of this step is that our errors 
could be correlated within districts, we cluster our standard errors at the district level.) 


To understand the reasoning behind this combination of techniques, it is helpful to see how 

the relationships between district-by-grade-level charter market share and average ELA and 
math achievement change as each technique is introduced. Accordingly, Figures A.1-A.2 show 
the relationship between district-by-grade-level charter market share among black students 
and average ELA and math achievement for black students with no statistical controls except for 
grade-level “dummies” (which ensure that we are comparing students in grade 3 to other third 
graders, as opposed to seventh graders). 


Per the figures, without the appropriate controls, this relationship is negative, which makes 
sense, given that charters enroll a disproportionate percentage of low-income students. And 
consistent with that interpretation, controlling for student demographics at the district-by- 
grade level reduces the negative slopes of our splines (Figures A.3—A.4). 
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Figures A.1—A.2. Association between "black charter market share" in urban 
districts and black ELA and math achievement with no statistical controls. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots). Sample includes all urban 
units with 0 to 50 percent black charter market share and average black enrollment >100 between 2009 and 2015. 


Figures A.3-A.4. Association between "black charter market share" in urban 
districts and black ELA and math achievement with demographic controls. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics. Sample includes all urban units with 0 to 50 percent black charter market share and average black enrollment >100 
between 2009 and 2015. 
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Intuitively, since our estimates are less negative when demographic controls are included, it 
seems likely that any “unobservable” characteristics are still working against charters. (For 
example, we can control for the percentage of students who are eligible for federal lunch 
subsidies, but not the depth of poverty that families below the subsidy-eligibility threshold 
experience.) However, by substituting district-by-grade-level fixed-effects for grade-level 
fixed-effects, we can account for such unobservable differences between district-by-grade-level 
units insofar as they are “time-invariant"—that is, insofar as the differences between black third 
graders in San Francisco and black third graders in Detroit remained constant over the course of 
the study period (Figures A.5—A.6). 


Figures A.5—A.6. Association between "black charter market share" in urban 
districts and black ELA and math achievement with unit and year fixed-effects. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, as well as district-by- grade-level unit and year fixed-effects. Sample includes all urban units with 0 to 50 percent 
black charter market share and average black enrollment >100 between 2009 and 2015. 


Per the figures, taking this step has a dramatic effect on the slope and interpretation of our 
graphs, suggesting that any unobservable and time-invariant differences between units were 
indeed working against charters rather than for them. However, because our unit fixed-effects 
cannot account for time-variant differences between district-by-grade-level units, our estimates 
could still be biased by unobserved trends insofar as these are not fully captured by our 
demographic controls. Accordingly, in our preferred model (which is the basis for the report), 
we include district-specific quadratic time-trends, in addition to district-by-grade-level-specific 
dummy variables (Figures A.7—A.8). 
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Figures A.7—A.8. Association between "black charter market share" in urban 
districts and black ELA and math achievement with district-specific time-trends. 
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Note: Figures show the predicted changes in the average ELA and math achievement of grade 3-8 black students in charter and 
traditional public schools as district-by-grade-level charter market share increases among black students. Estimates were 
generated using a cubic spline with knots at the 5th, 35th, 65th, and 95th percentiles (see red dots) and controlling for student 
demographics, district-by-grade-level unit and year fixed-effects, and district-specific quadratic time-trends. Sample includes all 
urban units with 0 to 50 percent black charter market share and average black enrollment >100 between 2009 and 2015. 


Per the figures, taking this final step makes many of our graphs straighter and more intuitive, 
suggesting that districts were trending in different directions for different ranges of black 
charter market share. And once again, our confidence intervals are reduced, suggesting that 
the “unobserved” component of these trends was non-trivial and allowing us to characterize 
the academic gains associated with higher black charter market share more confidently and 
precisely. 


Notably, controlling for per-pupil expenditures has little effect on the slopes of our graphs 
(though it does increase the size of our confidence intervals). Similarly, our results are robust 

to the exclusion of “percent special education,” which could also be affected by higher charter 
market share. And in many cases, they are robust to the exclusion of district- specific time-trends 
(though we believe these are advisable). Finally, although the skewed distribution of “charter 
market share” means we are relying on fewer districts and/or units as market share increases, 
our estimates are not sensitive to the number of “knots” in our splines. 


As discussed in Appendix B, our preferred estimates are broadly consistent with CREDO’'s 
estimates for the time period in question, giving us a useful (if rough) form of cross-validation. 
Furthermore, the assumptions on which the model is based are deeply intuitive: Different 
school districts are almost certainly on different achievement trajectories for reasons that have 
nothing to do with charters. But it’s unlikely that the same can be said of district-by-grade- 

level units. And controlling for district-level trends doesn't prevent us from using the within- 
district variation in district-by-grade-level charter market share and achievement (relative to the 
district-level trend) to estimate the former's likely contribution to the latter. 
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Admittedly, even after taking all of these steps, we cannot fully account for the effects of 
interdistrict student mobility—especially in smaller urban districts and for lower levels of charter 
market share. However, if the gains we attribute to higher charter market share were driven by 
the unobserved movement of high-achieving charter students across district lines, we would 
expect the slopes of our graphs to decrease as district size increases and the contribution of 
“interdistrict school choice” to average achievement decreases (as it must, unless students 

are more likely to commute to the center of large districts than the center of small districts). 
However, per the findings, this is actually the opposite of what we observe, even when we 
restrict the sample to urban units, where we would expect student mobility to play a larger role. 


Although it's impossible to prove definitively without student-level data, the simplest and most 
likely explanation for this pattern (and the other patterns we observe) is that higher charter 
market share in black and Hispanic communities has a positive impact on black and Hispanic 
students’ overall ELA and math achievement, consistent with the prior research on charters’ 
relative performance. 
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APPENDIX B 


ARE TRADITIONAL PUBLIC SCHOOLS IMPROVING? 


Although we can't disentangle the effects of charter competition from the effects of actually 
attending a charter school, most of our estimates of charters' impact on overall achievement 
imply a neutral (or perhaps slightly positive) effect on their district neighbors when viewed 
alongside CREDO's estimates of their relative performance during the time period in question. 


For example, in its 2015 study of charter performance in forty-one urban regions, CREDO 
estimated that black students in urban charters gained twenty-six days of learning in ELA 

and thirty-six days of learning in math per year,”* implying that over a period of five years 

(e.g., between kindergarten and fifth grade, which is the average grade level for the units in 
our sample) they should gain 130 days of learning in ELA and 180 days of learning in math— 

or roughly 0.7 and 1.0 grade levels—which is theoretically enough to boost average black 
achievement in urban districts with 50 percent charter market share by 0.35 and 0.5 grade levels 
in ELA and math, respectively.”” 


As this thought exercise demonstrates, many of our estimates of charters’ effect on overall 
achievement recall prior work on their relative performance. However, an implicit assumption of 
the arithmetic in the previous paragraph is that students are receiving most or all of the higher 
charter market share “treatment” for multiple years. In other words, it assumes that an increase 
in black or Hispanic charter market share in one grade is preceded by similar increases in lower 
grades. And of course, this assumption is frequently violated in reality—for example, when a 
charter school opens several grades simultaneously or whenever a middle school charter that is 
not affiliated with an elementary school opens its doors. 


Obviously, one potential implication of this line of reasoning is that we are underestimating 

the benefits of higher charter market share by averaging across all grade levels (3-8) instead 

of limiting ourselves to those where achievement gains are most likely to accumulate. (For 
example, the gains associated with higher black and Hispanic charter market share in small 
urban districts are clearer for grades 5-6 than for grades 3—4 or 7-8.) However, another potential 
implication is that some of the gains we observe are simply too large to explain without some 
sort of “competitive” or “spillover” effect on traditional public schools. 
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APPENDIX C 


LARGE URBAN AND "SUBURBAN" DISTRICTS 


Table C.1. Urban districts with at least 2500 black students per grade level (2009-2015) 


: ee: Average Black Charter 
Geographic School District Market Share (%) 


Atlanta Public Schools 1SES 
Baltimore City Public Schools 18.6 
Charlotte-Mecklenburg Schools 4.5 
Chicago Public Schools 12.9 
Clark County School District 5.6 
Cleveland Metropolitan School District 26.8 
Columbus School District 22.6 
Dallas Independent School District 16.7 
Davidson County Schools 8.7 
Detroit Public Schools Community District 38.4 
District of Columbia Public Schools 42.1 
Duval County Public Schools 5.5 
East Baton Rouge Parish School System 12.0 
Guilford County Schools 3.0 
Houston Independent School District 20.1 
Los Angeles Unified School District 23.5 
Memphis City School District Oks 
Milwaukee Public Schools 20.0 
New York City Public Schools nes, 
The School District of Philadelphia 28.6 
Wake County Public School System Al 
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Table C.2. Urban districts with at least 2,500 Hispanic students per grade level 
(2009-2015) 


Average Hispanic Charter 


Geographic School District 


Market Share (%) 
Albuquerque Public Schools US 
Austin Independent School District 7.5 
Bakersfield City School District 0.6 
Brownsville Independent School District 6.3 
Chicago Public Schools UD 
Clark County School District 1.5 
Dallas Independent School District 10.8 
Denver Public Schools 12.8 
Edinburg Consolidated Independent School District 6.1 
El Paso Independent School District 3.0 
Fort Worth Independent School District 1.6 
Fresno Unified School District 3: 
Houston Independent School District 16.4 
Long Beach Unified School District 1.6 
Los Angeles Unified School District 12.8 
New York City Department of Education 4.0 
North East Independent School District 4.3 
Northside Independent School District 3:7 
Philadelphia City Independent School District 19.5 
San Antonio Independent School District 22.0 
San Bernardino City Unified School District 41 
San Diego Unified School District 15.6 
Santa Ana Unified School District 4.0 
Socorro Independent School District 0.8 
Tucson Unified School District 10.8 
United Independent School District 2.4 
Ysleta Independent School District 2.6 
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Table C.3. Suburban districts with at least 2,500 Hispanic students per grade level 
(2009-2015). 


Average Hispanic Charter 


Geographic School District Market Share (%) 


Aldine Independent School District (Greater Houston) 


Broward County Public Schools (Greater Miami) 14.7 
Chula Vista Elementary School District (Greater San Diego) 20.4 
Cypress-Fairbanks Independent School District (Greater Houston) 1.9 
Gwinnett County Public Schools (Greater Atlanta) 0.5 
Hillsborough County Public Schools (Greater Tampa) 4.9 
Miami-Dade County Public Schools (Greater Miami) 14.8 
Orange County Public Schools (Greater Orlando) 2.4 
Palm Beach County School District (Greater Miami) 4.4 
Pharr-San Juan-Alamo Independent School District (Greater McAllen) 7.5 
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24. Since research shows that closure is highly concentrated among the lowest- performing charter 
schools, the charters that ultimately replace these schools are almost certainly higher performing 
on average. Center for Research on Education Outcomes. Lights Off: Practice and Impact of 
Closing Low-Performing Schools (San Francisco: Stanford, 2017), 4. https://credo.stanford.edu/ 
publications/school-closure-study-2017. 


25. Although the literature on interdistrict student mobility suggests that mobility rates are higher 
in smaller and urban areas and among low-income students—and by extension, in and between 
charter schools—because they enroll only a small fraction of students in most communities, it's 
likely that charters account for just a modest percentage of total interdistrict mobility, especially 
when “districts” are defined geographically. However, in the absence of student-level data, higher 
rates of “churn” in smaller and more urban districts may make it more difficult for the benefits 
or costs of charters to accumulate across grade levels and within cohorts. Powers, Jeanne, Amy 
Marcetti Topper, and Amanda Potterton. “Interdistrict mobility and charter schools in Arizona: 
Understanding the dynamics of public school choice.” Journal of Public Management and Social 
Policy 25, no. 3 (Fall 2018): 56-87. 


26. Center for Research on Education Outcomes. Urban Charter School Study Report on 41 Regions. 
San Francisco: Stanford, 2015. https://urbancharters. stanford.edu/download/Urban%20 
Charter%20School%20Study%20Report%200n%2041%20Regions.pdf. 


27. Although the appropriate conversion rate between “days of learning” in CREDO and “years of 
learning” in SEDA isn't clear, 180 days per year seems like a reasonable placeholder. 
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